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STATISTICAL  DESIGN  OF  EXPERIMENTS  FOR  CONTINUOUS  DATA 

Paul  C.  Cox 

Reliability  and  Statistics  Office 
The  Army  Missile  Test  and  Evaluation  Directorate 
White  Sands  Missile  Range,  New  Mexico 

I.  INT RO DU CT ION . I wish  to  talk  on  the  subject  of  how  one  would 
design  an  experiment  and  analyze  the  data  when  the  results  come  in  the 
form  of  a continuous  curve,  rather  than  just  a single  value.  This  is  an 
area  that  would  appear  to  have  extensive  application  in  science  and 
engineering.  For  example:  velocity  data,  trajectory  data,  meterological 
data,  thrust  data,  etc.  As  I just  mentioned,  I am  interested  in  the  design 
of  experiments,  which  means  I am  not  concerned  with  the  evaluation  of  a 
single  curve,  but  many  curves  obtained  as  a result  of  testing  under  several 
sets  of  conditions,  and  very  likely  each  set  of  conditions  will  have  some 
replications. 

To  illustrate  what  I have  in  mind,  I will  use  rocket  motor  thrust  curves, 
although  I could  have  used  some  other  type  of  curve  equally  effectively. 

Now  many  results  from  rocket  motor  tests  can  easily  be  analyzed.  For 
example:  average  exhaust  velocity;  effective  (average)  pressure;  total 
impulse;  specific  impulse,  etc.  These  are  simple  to  analyze  because  the 
data  for  a given  test  usually  comes  in  the  form  of  one  single  number. 

However,  if  we  want  to  estimate  a typical  or  average  thrust  curve  when  a 
motor  is  tested  under  given  conditions,  this  is  quite  a different  problem. 

To  keep  this  report  unclassified,  the  thrust  data  which  will  be  dis- 
cussed will  be  completely  fictitious.  The  data  is  not,  to  my  knowledge, 
appropriate  for  any  existing  rocket  motor,  but  the  general  shape  of  the 
curve  is  similar  to  what  may  be  expected  for  a number  of  motors  currently 
in  use. 

The  extensive  information  available  in  such  areas  as:  Regression 

Analysis;  Random  Processes;  Power  Spectral  Analysis;  Time  Series; 

Analysis  of  Covariance;  Multivariate  Analysis;  and  similar  fields  may 
easily  cover  the  problem  I am  going  to  present.  Therefore,  my  first 
question  is,  if  the  solution  is  readily  available  in  the  literature,  I would  (1) 
like  some  references.  My  second  question  is,  if  it  is  not  readily  available 
but  you  know  the  answers,  I will  appreciate  the  information.  (Please  note  (2) 
that  there  are  numbers  along  the  margins  of  this  report.  These  numbers 
refer  to  specific  questions  which  may  be  found  at  the  end  of  this  report.  ) 

I will  conclude  the  introduction  by  stating  that  the  five  panel  members, 

W.  T.  Federer,  B.  G.  Greenberg,  M.  A.  Schneiderman,  H.  L.  Lucas, 
and  H.  O.  Hartley  have  all  prepared  and  forwarded  their  comments.  These 
are  included  at  the  end  of  the  report  in  the  order  in  which  they  were  received. 
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II.  BASIC  ASSUMPTIONS. 

A.  We  will  begin  by  assuming  that  the  thrust  curves  to  be  considered 
in  this  study  will  look  something  like  the  one  illustrated  in  Figure  1,  although 
an  actual  curve  will  be  somewhat  more  irregular  than  the  example.  The 
letters  along  the  curve  will  represent  the  points  which  we  will  consider 
critical,  and  we  will  refer  to  these  points  many  times  in  the  future. 


Figure  1 

A Typical  Thrust  Curve  for  This  Study 


B.  The  Second  assumption  we  will  make  is  that  the  propellant  mix 
from  whence  the  motors  are  selected  as  well  as  the  preconditioning 
temperature  can  influence  the  shape  of  the  thrust  curve.  We  will  use  motors 
selected  from  the  three  propellant  mixes  (A,  B,  and  C),  and  conditioned 
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at  three  temperatures  (0°,  50°,  and  100°  F).  We  will  use  27  motors,  nine 
randomly  selected  from  each  mix,  and  of  the  nine,  three  conditioned  at  each 
of  the  temperatures.  This  arrangement  is  illustrated  in  Figure  2. 


Conditioning  Temperature 
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o° 

50° 
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3 

3 

3 
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3 

3 

3 
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3 

. 

3 

3 

Figure  2 

Number  of  motors  selected  from  each  mix 
and  conditioned  at  each  temperature. 


C.  The  third  assumption  is  that  temperature  conditioning  will  have  an 
effect  similar  to  that  illustrated  in  Figure  3.  These  three  curves  actually 
represent  the  average  of  the  curves  selected  from  Mix  A and  conditioned 
according  to  the  specified  temperature. 

D.  The  fourth  and  final  assumption  is  that  the  Test  Engineer  will  want 
the  following  information: 

1.  Does  Mix  difference  or  temperature  conditioning  have  a significant 
effect  upon  the  shape  of  a thrust  curve  ? 

2.  For  a given  propellant  mix  or  temperature  what  is  the  mean  or 
expected  thrust  curve  ? 

3.  In  addition  to  the  mean  thrust  curve,  confidence  and  tolerance 
bounds  are  desired. 

III.  ANALYSIS  OF  VARIANCE, 

A.  If  you  refer  again  to  Figure  1,  you  will  observe  a slightly  de- 
clining plateau  between  points  C and  D,  I resolved,  first  of  all,  to  compare 
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performance  to  this  plateau  area.  One  problem  is  evident  from  Figure  2, 
namely  the  plateau  areas  are  of  different  length  at  different  temperatures. 

I therefore  decided  to  study  only  the  region  from  15  to  42  seconds  inclusive. 
For  all  27  rounds,  I read  the  values  at  3 second  intervals,  that  is  to  say 
(15,  18,  . , . 42  seconds).  The  variances  appeared  to  be  homogeneous  in 
the  region,  so  we  performed  the  analysis  of  variance  illustrated  in  Fig.  3. 


Source  of  Variance 

d/f 

SS 

F 

Mix 

2 

43217.  71 

21608..  89 

82. 72** 

Temperature 

2 

488387. 22 

244193.  61 

934.78** 

Time 

9 

12325.  61 

1369. 50 

5. 24** 

(tyIix)(Temperature) 

4 

19794. 61 

4948. 65 

18.94** 

(Mix)(Time) 

18 

513.  70 

28.  52 

O.  11 

(Time)(Temperature) 

18 

2954. 70 

164.  15 

O.  63 

(Mix)(Time)(Temperature) 

36 

24459. 88 

679.44 

2.  60** 

Error 

Total 

180 

269 

47022.04 
638674.  97 

261.  23 
= (16.  16; 

^Significant  at  .05  level  (Single  and  double  asterisk  used  in  later  tables 
^Significant  at  .Ol  level  have  the  same  meaning  indicated  here.  ) 

Figure  4 

Analysis  of  Variance  of  Thrust  Values  between  15  and  42  seconds 

for  27  motors 


From  Figure  4 it  appears  that  for  these  curves,  temperature  has  a 
highly  significant  effect,  mix  has  an  important  effect,  and  there  is  a 
significant,  downward  trend  during  this  time  interval.  While  the  mix- 
temperature  interaction  is  significant,  its  F value  is  relatively  small;  so 
we  will  assume  it  is  not  really  critical.  It  may  also  be  observed  that  the 
- pooled  estimate  of  variance  is  261.  23,  and  we  will  therefore  assume  a 

standard  error  of  16  lbs.  with  180  degrees  of  freedom.  I then  proceeded 
to  construct  9 graphs,  one  for  each  combination  of  temperature  and  mix. 
Figure  5 illustrates  the  graph  for  Mix  B and  50  temperature.  I took  the 
sample  of  three  and  plotted  the  mean  values  for  the  interval  from  15  to  42 
seconds.  Next,  I used  the  pooled  estimate  of  variance  and  plotted  95*/fc> 
confidence  bounds  on  this  curve,  and  then  on  the  outside  of  this,  I plotted 
tolerance  bounds,  y = 95/o,  P = 90  fo. 

• 
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Clearly,  the  question  at  this  point  is  that  of  the  propriety  of  arbitrarily 
taking  a time  curve,  observing  the  values  at  stated  intervals  (every  three 
seconds  in  this  case)  and  considering  time,  along  with  mix  and  temperature,^ 
as  one  of  the  treatments  in  the  analysis  variance,  It  is  interesting  that  the 
error  term  has  180  degrees  of  freedom,  Had  we  arbitrarily  chosen  2 
second  intervals,  for  example,  instead  of  3 second  intervals  for  taking  our 
readings  the  degrees  of  freedom  for  error  would  have  increased  to  270. 

There  is  clearly  something  illogical  at  this  point. 

B.  Referring  again  to  Figure  1,  you  will  note  that  I have  arbitrarily 
selected  six  critical  points.  I then  proceeded  by  performing  an  analysis  of 
variance  for  both  the  X and  Y component  for  each  of  these  critical  compon- 
ents. The  Analysis  of  Variance  for  the  Y component  of  A is  given  in  Fig.  6 
and  the  X component  in  Figure  7. 
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d/f 

MS 

F 
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2 

99 
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Temp 

27, 746 

2 

13,  873 

104.  30 
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200 

4 

50 

Q.  38 

Error 

2,  398 

18 

133 
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30,  542 

26 

Figure  6 

Analysis  of  Variance  for  the  Y (Thrust  in  lbs)  Component  at  Point  A. 
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2 
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. 3129 

4 

. 1564 

4.48* 

Error 

. 6317 

18 

.0351 

T otal s 

1.  9052 

26 

Figure  7 

Analysis  of  Variance  for  the  X (Time  in  seconds)  Component  for  Point  A 
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From  this  it  may  be  observed  that  temperature  had  a significant  effect 
but  mix  did  not.  Returning  to  Figure  5,  the  mean  value  was  located  for 
Point  A and  a confidence  rectangle  was  drawn  about  it.  This  procedure  was 
also  followed  for  the  other  five  critical  points  and  these  points  were  then 
connected.  For  the  time  component,  temperature  had  a significant  effect 
for  all  six  critical  points,  mix  at  points  D,  E,  and  F.  For  the  thrust 
Component,  temperature  had  a significant  effect  at  all  points  except  F and 
mix  had  a significant  effect  at  points  B,  D,  and  E. 

One  will  obviously  be  concerned  at  this  point  by  the  fact  that  the  six 
critical  points  were  arbitrarily  chosen  and  are  not  precisely  defined.  This 
could  easily  result  in  considerable  inaccuracy  in  collecting  data  for  these 
points.  However,  this  fact  will  not  necessarily  be  emphasized  since  it  is 
not  really  relevant  to  the  basic  purpose  of  this  paper. 

However,  the  matter  of  performing  separate  analysis  of  the  time  and 
thrust  components  of  each  critical  point  is  highly  questionable,  and  I am 
certain  that  a procedure  applying  the  bivariate  normal  distribution  would 
be  in  order. 

Referring  either  to  Figures  1 or  5,  it  would  appear  reasonable  that  if 
an  analysis  of  variance  is  appropriate  for  C-D,  then  it  would  probably  be 
equally  appropriate  for  A-C  and  possibly  for  E-F.  In  fact,  it  would  appear 
more  sensible  than  attempting  to  locate  and  evaluate  critical  points. 

IV.  REGRESSION  ANALYSIS.  A second  approach,  and  one  which  I feel 
offers  more  promise  is  in  the  area  of  regression  analysis  and  polynomial 
fitting.  I will  discuss  a few  ideas  along  this  line  at  the  present  time. 

If  you  will  refer  to  Figure  1 again,  .1  arbitrarily  broke  the  graph  up  into 
four  distinct  segments.  These  are:  O-A;  A-C;  C-D;  and  E-F.  Then,  for 
all  27  motors,  I fitted  the  most  appropriate  polynomial,  that  is  to  say,  I 
fitted  a cubic  to  A-C  and  straight  lines  to  the  other  three  segments. 

I will  discuss  the  procedures  I followed  in  analyzing  segment  C-D  and 
state  little  more  than  that  analyses  were  performed  on  the  other  segments, 
and  upon  completion  all  segments  were  plotted  until  they  intersected.  Using 
the  values  from  15  to  45  seconds  and  recording  the  data  at  3 second  inter- 
vals, a straight  line  was  fitted  for  all  27  sets  of  data  and  an  analysis  of 
variance  was  performed  for  a,  b,  and  r in  the  equation  Y = a + b.  (X-30). 
Figure  8 gives  the  analysis  of  variance  and  mean  values  for  a,  while  Figure 
9 gives  the  analysis  of  variance  and  means  values  for  b,  (note  that  mix  had 
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no  significant  effect  upon  b,  so  the  mean  values  reflect  only  temperature).  (6) 
Neither  mix  nor  temperature  had  any  significant  effect  upon  r (the  correlation 
coefficient),  but  the  mean  value  of  the  27  correlation  coefficients  was  82  fo. 
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d/f 

MS 

F 
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4411 
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2206 
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Temperature 

45959 

2 

22980 

65.42** 

Interaction 
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4 
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1.  39 

Error 

6323 

18 

351 
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26 

Mix 

r 
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o° 

50° 

ioo° 

A 

280 

348 

37  5 

B 

292 

365 

385 

C 

264 

309 

376 

Ave 

278 

340 

379 

Figure  8 

The  .Analysis  of  Variance  and  the  Mean  Values  for  a, 
when  Fitting  the  Equation,  Y = a + b • (X-30)  . 

15  sec  <.  x < 45  sec  (all  means  computed  from  a sample  of  3, 
y = thrust  in  lbs.  , x = time  in  seconds). 
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Sources  of  Variation 


Mix 

Temperature 

Interaction 

Error 

Total 


MS 

F 

O.  319 
1.844 
O.  242 
O.  297 

1.040 
6.  209** 
. 815 

Temperature 

, 

o° 

O 

0 

ioo° 

Mean  Value  b 

-.410 

881 

-1.315 

Figure  9 

The  Analysis  of  Variance  and  Mean  Values  for  b, 
obtained  from  fitting  the  equation  Y ~ a + b-  (X-30), 

15  sec.  < x < 45  sec.  (all  means  computed  from  a sample  of  9) 


The  data  in  Figure  9 indicates  that  "b”  increases  almost  linearily  with 
temperature.  In  fact,  the  formula  b = -.410  - (.00905)  temp,  might  serve 
as  a guide  for  selecting  "b"  in  the  region  O < temp  < lOO  . If  one  desires 
a formula  for  estimating  "a"  in  the  region  for  O < tgmp  < lOO  , he  might 
try  the  formula:  a = 278  + 1.4  • (temp)  - . 004l(temp)  which  averages  out 
the  mix  effect. 

Again  referring  to  Figures  8 and  9,  one  may  observe  that  the  standard 
error  for  nan  is  18.  7 lbs.  with  18  degrees  of  freedom,  and  the  standard 
error  for  "b"  is  O.  545,  also  with  18  degrees  of  freedom. 

In  addition  to  this,  each  time  a line  is  fitted  by  least  squares,  it  is 
possible  to  obtain  a standard  error  of  estimates  and  standard  errors  for 
"a"  and  nb".  For  the  27  curves,  I pooled  these  standard  errors  and  obtain 
the  following  results: 

Pooled  Standard  Errors  of  estimates; 

3.40  lbs.  with  243  d/f. 


Pooled  Standard  Error  for  "b" : 
O.  324  with  243  d/f. 
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Pooled  Standard  Error  for  ’’a11 ; 

1.025  with  243  d/f. 

As  may  be  expected,  the  estimates  for  the  standard  error  for  both  "a”  and 
Mb"  are  larger  in  Figures  8 and  9 than  the  pooled  estimates  listed  above. 
This  is  reasonable  since  the  estimates  in  Figures  8 and  9 include  the  dis- 
persions that  exist  among  curves  from  the  same  lot  and  conditioned  at  the 
same  temperature,  while  the  pooled  estimates  reflect  the  variation  within 
only  a single  curve. 

Inasmuch  as  I am  attempting  to  classify  any  curves  which  come  from  a 
given  mix  and  a given  temperature,  it  would  seem  more  appropriate  to  use 
the  estimates  of  variability  in  Figures  8 and  9.  One  other  argument  for  this 
lies  in  the  fact  that  when  the  standard  error  of  estimates  was  computed  for 
each  of  the  27  curves,  it  was  computed  from  11  points,  selected  from  the 
thrust  curves  at  3 second  intervals.  15  sec  < time  <.45  sec.  Again  the 
question  arises  concerning  the  arbitrariness  in  choosing  3 second  intervals 
instead  of  some  other  intervals. 

Now  confidence  bounds  for  a regression  line  at  a point  x.  may  be 
computed  from  the  formula  1 

Y.  - t • S(Y.)  < E(Y.)  <T  + t • S(Y;) 


where  Y.  = a + b x. 

l l 


and  S^(Y.)  = S2(a)  + x2  • S2(b);  x.  = O. 

l l l 


Since  each  "a"  represents  an  average  of  3 numbers  and  each  "b"  represents 
an  average  of  9 numbers,  we  have: 


S2(Y.)  = — + x2 
l 3 l 


. 297  2 

= 117  + .033  x 
9 x 


These  ideas  are  illustrated  in  Figure  lO,  in  which  we  consider  Mix  C 
and  a temperature  of  O , and  fit  the  curve  from  15  sec  < X < 45  sec. 

(Segment  C-D).  This  curve  is  given  by  Y = 264  - .410  X.  or  Y=264- . 410- (X-30), 
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and  the  variance  S (Y)  = 117  + .033  was  used  to  compute  a confidence 
bound  about  the  curve.  Incidentally,  the  confidence  bound  in  Figure  lO  is 
very  close  to  the  one  illustrated  in  Figure  5. 

The  procedures  for  fitting  the  segments  (O-A)  and  (E-F)  could  be  quite 
similar  to  that  of  fitting  the  segment  (C-D).  In  fact  the  segment  (O-A) 
should  be  even  simpler.  To  fit  the  segment  (A-C),  it  is  suggested  that  a 
cubic  equation  be  fitted,  using  data  points  at  one  second  intervals.  It  is 
further  suggested  that  orthogonal  polynomials  be  used  when  fitting  a cubic  . . 
or  higher  degree  equation  to  simplify  the  process  of  obtaining  the  variance^ 
and  confidence  bounds. 

CONCLUSIONS . Frequently  it  i&  desired  to  design  an  experiment  when 
the  results  of  the  test  are  a continuous  curve  rather  than  a single  quantitative 
value.  Scientists  and  Engineers  frequently  want  to  know  whether  certain 
levels  of  a given  treatment  will  have  a significant  effect  upon  the  curve 
obtained,  and  what  will  an  average  or  expected  curve  be  for  a given  set  of 
conditions . 

I have  made  a few  suggestions  based  largely  upon  analysis  of  variance 
or  regression  analysis.  I will  greatly  appreciate  comments  on  the  proposed 
solutions,  but  more  important,  I would  like  suggestions  for  better  approaches 
to  the  problem. 

QUESTIONS. 

1.  Has  the  problem  of  designing  an  experiment  when  the  results  come 
as  a continuous  curve  rather  than  a single  value  ever  been  solved?  If  so, 
are  useful  references  available  ? 

2.  Do  you  have  any  ideas  of  additional  approaches  beside  those  sug- 
gested in  the  paper? 

3.  When  studying  a section  of  the  curve  such  as  C-D,  is  there  any 
justification  in  arbitrarily  selecting  a set  of  times  between  C and  D,  com- 
puting the  thrust  at  each  of  these  times  for  all  available  curvex,  and  per- 
forming an  analysis  of  variance  similar  to  that  given  by  Figure  4?  Perhaps 
this  would  be  in  order  with  certain  changes  in  procedure. 

4.  What  procedures  would  you  suggest  when  attempting  to  locate  a 
point  in  terms  of  both  its  X and  Y components  and  then  obtaining  both  a 
confidence  and  tolerance  region  about  this  point? 
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5.  Is  the  analysis  of  variance  for  a,  b,  and  r,  as  illustrated  in 
Figures  8 and  9 appropriate? 

6.  Have  you  any  suggestions  regarding  the  validity  of  the  techniques, 
using  regression  analysis,  that  were  discussed  in  this  section? 


